Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DistributedAtomic.Worker implements sub_get as a call #837

Merged

Conversation

philipgiuliani
Copy link
Contributor

We have a distributed environment in which we are starting Pads on different nodes. We experienced this issue multiple times in this cluster. Not sure how this issue didn't come up yet, as it seems to be in the codebase for a whole year.

10:32:46.327 [error] GenServer #PID<50061.5059.0> terminating
** (FunctionClauseError) no function clause matching in Membrane.Core.Element.AtomicDemand.DistributedAtomic.Worker.handle_cast/2
(membrane_core 1.1.0) lib/membrane/core/element/atomic_demand/distributed_atomic/worker.ex:46: Membrane.Core.Element.AtomicDemand.DistributedAtomic.Worker.handle_cast({:sub_get, #Reference<50061.4261161875.3378380802.63969>, 150}, nil)
(stdlib 5.1.1) gen_server.erl:1103: :gen_server.try_handle_cast/3
(stdlib 5.1.1) gen_server.erl:1165: :gen_server.handle_msg/6
(stdlib 5.1.1) proc_lib.erl:251: :proc_lib.wake_up/3
Last message: {:"$gen_cast", {:sub_get, #Reference<50061.4261161875.3378380802.63969>, 150}}
10:32:46.682 [error] GenServer #PID<0.2944.0> terminating
** (stop) exited in: GenServer.call(#PID<50061.5059.0>, {:get, #Reference<50061.4261161875.3378380802.63969>}, 5000)
** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
(elixir 1.16.1) lib/gen_server.ex:1114: GenServer.call/3
(membrane_core 1.1.0) lib/membrane/core/stalker.ex:496: anonymous fn/1 in Membrane.Core.Stalker.scrape_metrics/1
(elixir 1.16.1) lib/enum.ex:1700: Enum."-map/2-lists^map/1-1-"/2
(elixir 1.16.1) lib/enum.ex:1700: Enum."-map/2-lists^map/1-1-"/2
(membrane_core 1.1.0) lib/membrane/core/stalker.ex:372: Membrane.Core.Stalker.do_handle_info/2
(membrane_core 1.1.0) lib/membrane/core/stalker.ex:366: Membrane.Core.Stalker.handle_info/2
(stdlib 5.1.1) gen_server.erl:1077: :gen_server.try_handle_info/3
(stdlib 5.1.1) gen_server.erl:1165: :gen_server.handle_msg/6
Last message: :scrape_metrics

@dmorn
Copy link

dmorn commented Jul 3, 2024

We noticed also that now the issue of the latest hotfix (2a545e3) is still present apparently:

s    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:100: Membrane.Core.Element.PadController.do_handle_link/6
    (membrane_core 1.1.0) lib/membrane/core/element.ex:187: Membrane.Core.Element.do_handle_call/3
    (membrane_core 1.1.0) lib/membrane/core/element.ex:174: Membrane.Core.Element.handle_call/3
    (stdlib 5.1.1) gen_server.erl:1113: :gen_server.try_handle_call/4
    (stdlib 5.1.1) gen_server.erl:1142: :gen_server.handle_msg/6
    (stdlib 5.1.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3


10:47:38.880 [warning] Failed to insert a metric into the observability ETS.
Error: ** (ArgumentError) errors were found at the given arguments:

  * 1st argument: the table identifier does not refer to an existing ETS table

    (stdlib 5.1.1) :ets.insert(#Reference<0.2642032877.967180289.249079>, {{:atomic_demand, ["<0.2788.0>/", "{:client, %{server: #PID<0.2788.0>, ref: #Reference<0.2642032877.967311361.96835>, translate_language_code: \"DE\", enable_voiceover: true, input_language_code: \"en\"}}/", ":voiceover"], :output}, {:__membrane_stalker_function_metric__, #Function<4.87660500/0 in Membrane.Core.Element.PadController.init_pad_data/8>}})
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:292: Membrane.Core.Element.PadController.init_pad_data/8
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:100: Membrane.Core.Element.PadController.do_handle_link/6
    (membrane_core 1.1.0) lib/membrane/core/element.ex:187: Membrane.Core.Element.do_handle_call/3
    (membrane_core 1.1.0) lib/membrane/core/element.ex:174: Membrane.Core.Element.handle_call/3
    (stdlib 5.1.1) gen_server.erl:1113: :gen_server.try_handle_call/4
    (stdlib 5.1.1) gen_server.erl:1142: :gen_server.handle_msg/6
    (stdlib 5.1.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3


10:47:38.880 [warning] Failed to insert a metric into the observability ETS.
Error: ** (ArgumentError) errors were found at the given arguments:

  * 1st argument: the table identifier does not refer to an existing ETS table

    (stdlib 5.1.1) :ets.insert(#Reference<0.2642032877.967180289.249079>, {{:total_buffers, ["<0.2788.0>/", "{:client, %{server: #PID<0.2788.0>, ref: #Reference<0.2642032877.967311361.96835>, translate_language_code: \"DE\", enable_voiceover: true, input_language_code: \"en\"}}/", "{:transcript, :tee}"], :input}, {:__membrane_stalker_function_metric__, #Function<3.87660500/0 in Membrane.Core.Element.PadController.init_pad_data/8>}})
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:286: Membrane.Core.Element.PadController.init_pad_data/8
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:188: Membrane.Core.Element.PadController.do_handle_link/6
    (membrane_core 1.1.0) lib/membrane/core/element.ex:187: Membrane.Core.Element.do_handle_call/3
    (membrane_core 1.1.0) lib/membrane/core/element.ex:174: Membrane.Core.Element.handle_call/3
    (stdlib 5.1.1) gen_server.erl:1113: :gen_server.try_handle_call/4
    (stdlib 5.1.1) gen_server.erl:1142: :gen_server.handle_msg/6
    (stdlib 5.1.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3


10:47:38.881 [warning] Failed to insert a metric into the observability ETS.
Error: ** (ArgumentError) errors were found at the given arguments:

  * 1st argument: the table identifier does not refer to an existing ETS table

    (stdlib 5.1.1) :ets.insert(#Reference<0.2642032877.967180289.249079>, {{:atomic_demand, ["<0.2788.0>/", "{:client, %{server: #PID<0.2788.0>, ref: #Reference<0.2642032877.967311361.96835>, translate_language_code: \"DE\", enable_voiceover: true, input_language_code: \"en\"}}/", "{:transcript, :tee}"], :input}, {:__membrane_stalker_function_metric__, #Function<4.87660500/0 in Membrane.Core.Element.PadController.init_pad_data/8>}})
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:292: Membrane.Core.Element.PadController.init_pad_data/8
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:188: Membrane.Core.Element.PadController.do_handle_link/6
    (membrane_core 1.1.0) lib/membrane/core/element.ex:187: Membrane.Core.Element.do_handle_call/3
    (membrane_core 1.1.0) lib/membrane/core/element.ex:174: Membrane.Core.Element.handle_call/3
    (stdlib 5.1.1) gen_server.erl:1113: :gen_server.try_handle_call/4
    (stdlib 5.1.1) gen_server.erl:1142: :gen_server.handle_msg/6
    (stdlib 5.1.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3


10:47:38.881 [warning] Failed to insert a metric into the observability ETS.
Error: ** (ArgumentError) errors were found at the given arguments:

  * 1st argument: the table identifier does not refer to an existing ETS table

    (stdlib 5.1.1) :ets.insert(#Reference<0.2642032877.967180289.249079>, {{:auto_demand_size, ["<0.2788.0>/", "{:client, %{server: #PID<0.2788.0>, ref: #Reference<0.2642032877.967311361.96835>, translate_language_code: \"DE\", enable_voiceover: true, input_language_code: \"en\"}}/", "{:transcript, :tee}"], :input}, {:__membrane_stalker_function_metric__, #Function<8.87660500/0 in Membrane.Core.Element.PadController.merge_pad_mode_data/4>}})
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:422: anonymous fn/2 in Membrane.Core.Element.PadController.merge_pad_mode_data/4
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:421: Membrane.Core.Element.PadController.merge_pad_mode_data/4
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:318: Membrane.Core.Element.PadController.init_pad_data/8
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:188: Membrane.Core.Element.PadController.do_handle_link/6
    (membrane_core 1.1.0) lib/membrane/core/element.ex:187: Membrane.Core.Element.do_handle_call/3
    (membrane_core 1.1.0) lib/membrane/core/element.ex:174: Membrane.Core.Element.handle_call/3
    (stdlib 5.1.1) gen_server.erl:1113: :gen_server.try_handle_call/4


10:47:38.882 [warning] Failed to insert a metric into the observability ETS.
Error: ** (ArgumentError) errors were found at the given arguments:

  * 1st argument: the table identifier does not refer to an existing ETS table

    (stdlib 5.1.1) :ets.insert(#Reference<0.2642032877.967180289.249079>, {{:total_buffers, ["<0.2788.0>/", ":core/", "{:transcript, :tee}"], {Membrane.Pad, :output, #Reference<0.2642032877.967311362.66038>}}, {:__membrane_stalker_function_metric__, #Function<3.87660500/0 in Membrane.Core.Element.PadController.init_pad_data/8>}})
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:286: Membrane.Core.Element.PadController.init_pad_data/8
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:100: Membrane.Core.Element.PadController.do_handle_link/6
    (membrane_core 1.1.0) lib/membrane/core/element.ex:187: Membrane.Core.Element.do_handle_call/3
    (membrane_core 1.1.0) lib/membrane/core/element.ex:174: Membrane.Core.Element.handle_call/3
    (stdlib 5.1.1) gen_server.erl:1113: :gen_server.try_handle_call/4
    (stdlib 5.1.1) gen_server.erl:1142: :gen_server.handle_msg/6
    (stdlib 5.1.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3


10:47:38.882 [warning] Failed to insert a metric into the observability ETS.
Error: ** (ArgumentError) errors were found at the given arguments:

  * 1st argument: the table identifier does not refer to an existing ETS table

    (stdlib 5.1.1) :ets.insert(#Reference<0.2642032877.967180289.249079>, {{:atomic_demand, ["<0.2788.0>/", ":core/", "{:transcript, :tee}"], {Membrane.Pad, :output, #Reference<0.2642032877.967311362.66038>}}, {:__membrane_stalker_function_metric__, #Function<4.87660500/0 in Membrane.Core.Element.PadController.init_pad_data/8>}})
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:292: Membrane.Core.Element.PadController.init_pad_data/8
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:100: Membrane.Core.Element.PadController.do_handle_link/6
    (membrane_core 1.1.0) lib/membrane/core/element.ex:187: Membrane.Core.Element.do_handle_call/3
    (membrane_core 1.1.0) lib/membrane/core/element.ex:174: Membrane.Core.Element.handle_call/3
    (stdlib 5.1.1) gen_server.erl:1113: :gen_server.try_handle_call/4
    (stdlib 5.1.1) gen_server.erl:1142: :gen_server.handle_msg/6
    (stdlib 5.1.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3


10:50:04.301 [error] retry: got exception, will retry in 1000ms, 3 attempts left
10:50:04.301 [error] ** (Req.TransportError) socket closed

@FelonEkonom
Copy link
Member

@philipgiuliani could you please update your branch with the master branch?
I will release this pull request in around 10 days in version 1.1.1, because currently I am on the vacations.
Thank you very much for your bug fix 👍 👍 👍

@philipgiuliani
Copy link
Contributor Author

I've rebased the branch onto master.

@FelonEkonom FelonEkonom merged commit 57ff6c9 into membraneframework:master Jul 24, 2024
3 of 4 checks passed
@FelonEkonom
Copy link
Member

@philipgiuliani your fix has been released in v1.1.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants